{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Return of Investments for online marketing\n",
    "\n",
    "## Try me\n",
    " [![Open In Colab](../../_static/colabs_badge.png)](https://colab.research.google.com/github/ffraile/operations-research-notebooks/blob/main/docs/source/Simulation/Solved/Return%20of%20search%20engine%20investments.ipynb)[![Binder](../../_static/binder_badge.png)](https://mybinder.org/v2/gh/ffraile/operations-research-notebooks/main?labpath=docs%2Fsource%2FSimulation%2FSolved%2FReturn%20of%20search%20engine%20investments.ipynb)\n",
    "\n",
    "The following metrics and indicators determine the expenditures and return of investments of marketing investments in paid inclusion (including your company's web page in search results) for your company:\n",
    "\n",
    "- **S (Total Search results in period):** Total number of times the keywords are used in searches in the planning period.\n",
    "- **CPM (Cost per 1000 impressions):** Cost of the search engine per every 1000 impressions, every time your website appears in a search result\n",
    "- **CPC (Cost per click):** Cost per every click\n",
    "- **CTR (Click-Through Rate):** rate between the number of clicks and number of impressions\n",
    "- **CVR (Conversion Rate):** rate between the number of visitors and the number of users\n",
    "\n",
    "With this, the number of visitors (V) and the number of users (U) can be calculated as:\n",
    "\n",
    "$U = V * CVR = S * CTR * CVR$\n",
    "\n",
    "The total costs ($C_t$) of the search engine investments are:\n",
    "\n",
    "$C_t = CPC*V + CPM/1000*S$\n",
    "\n",
    "\n",
    "The **ARPU (Average revenue per user in planning period)** is the average revenue per user, or the total revenue in the planning period (R), divided by the number of users:\n",
    "\n",
    "$ARPU = R/U$\n",
    "\n",
    "The Return of Investments for online marketing is calculated as:\n",
    "\n",
    "$ROI = \\frac{R - C_t}{C_t}$\n",
    "\n",
    "After some research, you have found that the different metrics can be modeled as normal distributions with the following parameters:\n",
    "\n",
    "| Metric | Mean | Standard Deviation |\n",
    "|--------|------|--------------------|\n",
    "|Search results | 600000 | 100000 |\n",
    "| CTR | 0.05 | 0.01 |\n",
    "| CVR | 0.5 | 0.1 |\n",
    "\n",
    "These are the search engine fees for your keywords:\n",
    "- CPM = 0,3€\n",
    "- CPC = 0,75€\n",
    "\n",
    "You estimate your ARPU in 5€\n",
    "\n",
    "**a.** Calculate the total costs, the total Return of Investments and the average cost per acquisition using Montecarlo Simulation\n"
   ]
  },
  {
   "cell_type": "markdown",
   "source": [
    "Ok, Let's first import the libraries we will need:"
   ],
   "metadata": {
    "collapsed": false
   }
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import numpy as np"
   ]
  },
  {
   "cell_type": "markdown",
   "source": [
    "Ok, now we can start with the simulation. First, we will define the number of simulations we want to run. We will use 100000 simulations, but you can change this number to see how the results change. Once, we define the number of simulations, we will also define the properties of the random variables we want to simulate with Montecarlo (mean and standard deviation). We will use the numpy library to generate the random variables. Finally, in this basic implementation, we will define a for loop to run the simulation and store the results in a dataframe. We will also calculate the ROI for each simulation and store it in the dataframe. Finally, We will use the describe function to get the statistics of the results."
   ],
   "metadata": {
    "collapsed": false
   }
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "outputs": [],
   "source": [
    "# Number of simulations\n",
    "num_simulations = 100000\n",
    "\n",
    "# Parameters of the revenue model\n",
    "#Data Search Results and search engines costs\n",
    "#Data Search Results and search engines costs\n",
    "search_results_avg = 600000\n",
    "search_results_std = 100000\n",
    "\n",
    "ctr_mean = 0.05\n",
    "ctr_std = 0.01\n",
    "\n",
    "cvr_mean = 0.5\n",
    "cvr_std = 0.1\n",
    "\n",
    "# Costs and unitary profit\n",
    "CPM = 0.3\n",
    "CPC = 0.75\n",
    "\n",
    "ARPU = 5\n",
    "\n",
    "# We initialize a numpy array with the number of simulations and the number of variables we want to simulate\n",
    "results = np.zeros((num_simulations, 8))\n",
    "\n",
    "# We run the simulation\n",
    "for i in range(num_simulations):\n",
    "    # First we calculate the total number of impacts or searches\n",
    "    total_impacts = np.random.normal(search_results_avg,search_results_std)\n",
    "\n",
    "    # Then we calculate the total number of visitors using the Click-Through Rate (CTR)\n",
    "    CTR = np.random.normal(ctr_mean, ctr_std)\n",
    "    total_visitors = total_impacts * CTR\n",
    "\n",
    "    # Then we calculate the total number of customers using the Conversion Rate (CVR)\n",
    "    CVR = np.random.normal(cvr_mean, cvr_std)\n",
    "    total_customers = total_visitors * CVR\n",
    "\n",
    "    # Finally, we calculate the total cost and the total revenues, using the costs and the ARPU\n",
    "    total_cost = CPC*total_visitors + CPM*total_impacts/1000\n",
    "    total_revenues = total_customers * ARPU\n",
    "    ROI = (total_revenues - total_cost)/total_cost\n",
    "\n",
    "    # We store the results in the numpy array, in row i\n",
    "    results[i,:] = [total_impacts, CTR, total_visitors, CVR, total_customers, total_cost, total_revenues, ROI]\n",
    "\n",
    "# We create a dataframe with the results\n",
    "df_results = pd.DataFrame(results, columns = ['Total impacts', 'CTR', 'Total visitors', 'CVR', 'Total customers', 'Total cost', 'Total revenues', 'ROI'])\n",
    "results.describe()"
   ],
   "metadata": {
    "collapsed": false
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "This code works and is easy to interpret, but it does not leverage one of the most outstanding features of Numpy, working with vectors and matrices. Let's see how we can improve the code using vectors and matrices, avoiding the for loop:"
   ],
   "metadata": {
    "collapsed": false
   }
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": "            impacts            CTR       Visitors            CVR  \\\ncount  1.000000e+05  100000.000000  100000.000000  100000.000000   \nmean   5.999030e+05       0.050052   30031.266831       0.499396   \nstd    1.001245e+05       0.010003    7901.093998       0.099647   \nmin    1.627489e+05       0.006749    3111.804727       0.095343   \n25%    5.327494e+05       0.043324   24461.139430       0.432389   \n50%    5.997867e+05       0.050051   29532.234129       0.499373   \n75%    6.674925e+05       0.056801   35082.456403       0.566536   \nmax    1.045158e+06       0.092695   73104.943275       0.958135   \n\n           Customers       Revenues           Cost            ROI  \ncount  100000.000000  100000.000000  100000.000000  100000.000000  \nmean    14997.328542   74986.642710   22703.421018       2.301716  \nstd      5015.323090   25076.615449    5945.011469       0.658838  \nmin      1425.791553    7128.957767    2472.171271      -0.369106  \n25%     11421.034780   57105.173899   18513.188951       1.858309  \n50%     14445.367567   72226.837833   22325.493179       2.301226  \n75%     18009.074903   90045.374513   26502.929706       2.745805  \nmax     47847.191767  239235.958836   55099.856282       5.337203  ",
      "text/html": "<div>\n<style scoped>\n    .dataframe tbody tr th:only-of-type {\n        vertical-align: middle;\n    }\n\n    .dataframe tbody tr th {\n        vertical-align: top;\n    }\n\n    .dataframe thead th {\n        text-align: right;\n    }\n</style>\n<table border=\"1\" class=\"dataframe\">\n  <thead>\n    <tr style=\"text-align: right;\">\n      <th></th>\n      <th>impacts</th>\n      <th>CTR</th>\n      <th>Visitors</th>\n      <th>CVR</th>\n      <th>Customers</th>\n      <th>Revenues</th>\n      <th>Cost</th>\n      <th>ROI</th>\n    </tr>\n  </thead>\n  <tbody>\n    <tr>\n      <th>count</th>\n      <td>1.000000e+05</td>\n      <td>100000.000000</td>\n      <td>100000.000000</td>\n      <td>100000.000000</td>\n      <td>100000.000000</td>\n      <td>100000.000000</td>\n      <td>100000.000000</td>\n      <td>100000.000000</td>\n    </tr>\n    <tr>\n      <th>mean</th>\n      <td>5.999030e+05</td>\n      <td>0.050052</td>\n      <td>30031.266831</td>\n      <td>0.499396</td>\n      <td>14997.328542</td>\n      <td>74986.642710</td>\n      <td>22703.421018</td>\n      <td>2.301716</td>\n    </tr>\n    <tr>\n      <th>std</th>\n      <td>1.001245e+05</td>\n      <td>0.010003</td>\n      <td>7901.093998</td>\n      <td>0.099647</td>\n      <td>5015.323090</td>\n      <td>25076.615449</td>\n      <td>5945.011469</td>\n      <td>0.658838</td>\n    </tr>\n    <tr>\n      <th>min</th>\n      <td>1.627489e+05</td>\n      <td>0.006749</td>\n      <td>3111.804727</td>\n      <td>0.095343</td>\n      <td>1425.791553</td>\n      <td>7128.957767</td>\n      <td>2472.171271</td>\n      <td>-0.369106</td>\n    </tr>\n    <tr>\n      <th>25%</th>\n      <td>5.327494e+05</td>\n      <td>0.043324</td>\n      <td>24461.139430</td>\n      <td>0.432389</td>\n      <td>11421.034780</td>\n      <td>57105.173899</td>\n      <td>18513.188951</td>\n      <td>1.858309</td>\n    </tr>\n    <tr>\n      <th>50%</th>\n      <td>5.997867e+05</td>\n      <td>0.050051</td>\n      <td>29532.234129</td>\n      <td>0.499373</td>\n      <td>14445.367567</td>\n      <td>72226.837833</td>\n      <td>22325.493179</td>\n      <td>2.301226</td>\n    </tr>\n    <tr>\n      <th>75%</th>\n      <td>6.674925e+05</td>\n      <td>0.056801</td>\n      <td>35082.456403</td>\n      <td>0.566536</td>\n      <td>18009.074903</td>\n      <td>90045.374513</td>\n      <td>26502.929706</td>\n      <td>2.745805</td>\n    </tr>\n    <tr>\n      <th>max</th>\n      <td>1.045158e+06</td>\n      <td>0.092695</td>\n      <td>73104.943275</td>\n      <td>0.958135</td>\n      <td>47847.191767</td>\n      <td>239235.958836</td>\n      <td>55099.856282</td>\n      <td>5.337203</td>\n    </tr>\n  </tbody>\n</table>\n</div>"
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "num_simulations = 100000\n",
    "\n",
    "#Data Search Results and search engines costs\n",
    "search_results_avg = 600000\n",
    "search_results_std = 100000\n",
    "\n",
    "ctr_mean = 0.05\n",
    "ctr_std = 0.01\n",
    "\n",
    "cvr_mean = 0.5\n",
    "cvr_std = 0.1\n",
    "\n",
    "# Costs and unitary profit\n",
    "CPM = 0.3\n",
    "CPC = 0.75\n",
    "\n",
    "ARPU = 5\n",
    "\n",
    "\n",
    "# Total number of impacts or searches, each row represents a simulation\n",
    "total_impacts = np.random.normal(search_results_avg,search_results_std, num_simulations)\n",
    "\n",
    "# Click-Through Rate, each row represents a simulation\n",
    "CTR = np.random.normal(ctr_mean, ctr_std, num_simulations)\n",
    "\n",
    "# Total clicks or visitors\n",
    "total_visitors = total_impacts * CTR\n",
    "\n",
    "# Conversion rate\n",
    "CVR = np.random.normal(cvr_mean, cvr_std, num_simulations)\n",
    "\n",
    "#Total customers or users\n",
    "total_customers = total_visitors * CVR\n",
    "\n",
    "#Total search engine costs\n",
    "total_cost = CPC*total_visitors + CPM*total_impacts/1000\n",
    "\n",
    "#Total revenues estimated from ARPU\n",
    "total_revenues = total_customers * ARPU\n",
    "\n",
    "#ROI (total revenues - total costs)/total cost\n",
    "\n",
    "ROI = (total_revenues - total_cost)/total_cost\n",
    "\n",
    "\n",
    "results_df = pd.DataFrame({'impacts':total_impacts, 'CTR':CTR, 'Visitors':total_visitors, 'CVR':CVR, 'Customers':total_customers, 'Revenues':total_revenues, 'Cost':total_cost, 'ROI':ROI})\n",
    "\n",
    "#Now we can see the statistics using the describe function:\n",
    "results_df.describe()\n"
   ]
  },
  {
   "cell_type": "markdown",
   "source": [
    "Note that the only difference is that now we have used the numpy library to generate bi-dimensional matrices with the random variables, and then just algebraic operations to calculate and store the results. This is much more efficient than using a for loop."
   ],
   "metadata": {
    "collapsed": false
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "Ok, now we can answer the questions just use the function hist() of the dataframe to plot the histogram:"
   ],
   "metadata": {
    "collapsed": false
   }
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": "<Figure size 432x288 with 9 Axes>",
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAX0AAAEICAYAAACzliQjAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8vihELAAAACXBIWXMAAAsTAAALEwEAmpwYAAAxzElEQVR4nO2deZwdRdnvvz8TNhNeCCTEkKDDqgRz1RDZXx0WISwK94LIIgREkRe4ygWXQVD84EJAg8Lrgii8hIiyuBGJGCMwKlECCVtIIGbAYBKyyBaYqEDIc/+oOpPO4ZyZM3OW7nP6+X4+/TndVdVVT9VT9XRt3UdmhuM4jpMP3pS2AI7jOE7jcKPvOI6TI9zoO47j5Ag3+o7jODnCjb7jOE6OcKPvOI6TI3Jt9CUtkNSethyO49SPStq5pLdK6pY0qDFSpUeujb6Z7WFmnWnLASCpTZJJGpy2LHlH0kmS5kYjsELSnZJ+H6+7Jb0q6bXE9Z0J/RXclkjqSDsveUDSbyVdWsL9aEkrgXf11c7N7O9mNtTMXo/3dkr6eH0kTpdcG33HKUbS+cC3ga8DI4G3At8DHopGYWj0u6VwbWaHJ6LYOoY5DviipA80Nge5ZCrwUUkqcj8FuMnM1jVSGAUya1szK1gjiL2xQyR9WdJtkn4s6WVJ8yXtJulCSaslLZV0aOK+TkmXSbpf0kuSbpe0TcL/NkkrJa2R9EdJeyT8tpA0RdLT0f9eSVsAf4xBXow9xX0l7SLpDzHcs5JuaVzp5A9JWwGXAueY2S/MbK2ZvWZmvzazz/YnLjObCywA3l0HUZ2N+RWwLfCfBQdJw4CjgBsL7Ty67xVHcS9JWiXpyujeM9KW9LUY13diW/xODLOfpAdie3xA0n6J9DolfU3SbOCfwE6STpP0VLQpf5N0coPKo1dybfSL+CAwDRgGPATMJJTPaIIh+EFR+FOBjwGjgHXA1Qm/O4Fdge2AB4GbEn7fBPYE9gO2AT4HrAfeF/23jr3HvwBfAX4XZRoD/HcN8umUZ19gc+CX1UYkaR/gnUBXtXE5vWNm/wJuJbTJAscDT5jZI0XBrwKuMrP/AHaO9xXHdxHwJ+Dc2BbPjZ26GYR2vi1wJTBD0raJW08BzgS2BP4Rwx5uZlsS2vvD1ea1FrjR38CfzGxmHAreBowAJpvZa8DNQJukrRPhp5nZY2a2FvgicHxhEcjMrjezl83sFeDLwLskbRWHfB8DPm1my83sdTP7cwxXiteAtwHbm9m/zezeOuTb2cC2wLNVTgc8K+lfwF8I00K/qoVgTp9MBY6TtHm8PjW6FfMasIuk4WbWbWb3VRj/kcBiM5tmZuvM7KfAE4TOYoEbzGxBrD/rCJ25d0rawsxWmNmCAeWsxrjR38CqxPm/CI3/9cQ1wNBEmKWJ86eBTYDhkgZJmizpSUkvAUtimOHx2Bx4skKZPgcIuD/uQPhYxblxBsJzBB1Ws5g+nFBPLgDaCfXCqTOxQ/QscIyknYG9gJ+UCHoGsBvwRJyiOarCJLYntPMkTxNmAgr02ITYGfwIcBawQtIMSe+oMK264kZ/4OyQOH8roQfxLHAScDRwCLAV0BbDKPr/mzCsLOYNnzs1s5Vm9gkz2x74JPA9SbvUKgPOG/gL8ApwTDWRxBHclQRdn10DuZzKuJHQw/8oMNPMVhUHMLPFZnYiYer1cuBnkoaUiKu4PT5DGHUneSuwvNw9cebgA4Qp4CeAH/YjL3XDjf7A+aiksZLeTJjz/1kcGWxJMBzPAW8m7PQAwMzWA9cDV0raPo4K9pW0GWEOcD2wUyG8pA9LGhMvXyBUqvUNyFsuMbM1wJeA70o6RtKbJW0i6XBJVwwgysnA5xJTDk59uZHQ2foEpad2kPRRSSNiW3wxOpdqU6tItEXgN8BuCtt5B0v6CDAWuKNMOiMVtowOIdiD7jLpNBw3+gNnGnADsJIwZfOp6H4jYdi3HFgIFM8ZfgaYDzwAPE/obbzJzP4JfA2YLenFuBD4XmCOpG5gOmEt4Kl6ZirvmNkU4HzgYsKDeClwLgObm59BeFh/olbyOeUxsyXAn4EhhPZSionAgtimrgJOiAvBxVxFWCN4QdLVZvYcYTfQBYQO3eeAo8zs2TLpvIlQj54htPP3A/81oIzVGPmfqPQfSZ3Aj83sR2nL4jiO0x+8p+84jpMj3Og7juPkCJ/ecRzHyRHe03ccx8kRmf6i4/Dhw62trS1tMTZi7dq1DBlSaltvc9CX/PPmzXvWzEY0Sp566bgRemrWNNLQ8YgRI5q63fSXtO1Erzo2s8wee+65p2WNe+65J20RqqIv+YG51gI6boSemjWNNHTc7O2mv6Sd39507NM7juM4OcKNfo1p65iRtghOnXEdNy+uOzf6NSVZobxytTZtHTOYv3yN69lpOtzo14hSBt8NQuvhOnWanUzv3nEcx6kF/rDegPf064xXttbHdZxtivXT1jEj1zpzo18D8lyBnIDXAadZcKPfANwgND+V9A5dz9nDdfJG3Og7jpNL8vpAcKPvOI6TI9zoV0mlvYW89iryhuu5ucijvtzoN5A8VjDHyTp5a5du9B3HaUnyZswrxY1+FQykUnlFbD5cZ82H66w8bvQHiFeqfOB6zgd50rMbfcepMXkyIE7z4UbfcRwnR7jRd5wyVNNj995+egy07POiMzf6AyAvlSPPuI6dVsWNfgq4QXEcJy3c6DtOnfCHu5NF3Og7jtMyVPugzcOD2o2+4xRRy4afByPiNBd9Gn1JO0i6R9JCSQskfTq6byNplqTF8XdYdJekqyV1SXpU0vhEXJNi+MWSJtUvW9knS8Zg6dKlHHjggYwdOxZgD9ex04zUqk1lqW3Wg0p6+uuAC8xsLLAPcI6ksUAHcJeZ7QrcFa8BDgd2jceZwPchGBDgEmBvYC/gkoIRaSbmL1+Ttgg1Z/DgwUyZMoWFCxcCPE7Odew4rUyfRt/MVpjZg/H8ZYJRGA0cDUyNwaYCx8Tzo4EbLXAfsLWkUcBhwCwze97MXgBmARNrmRlnYIwaNYrx43s66+txHdeULPQcfTTnFBjcn8CS2oD3AHOAkWa2InqtBEbG89HA0sRty6JbOffiNM4k9B4ZOXIknZ2d/RGx7ozcAi4Yt64mcaWRt+7u7r7S3ZQW0HEF+SxLpfrtT10YqCzV5CPJc889x4knnshuu+3GgQceWBjNzQJOI4zmJkvqIIzmPs/Go7m9CaO5vROjuQmAAfMkTY8P+dTIwoO1WajY6EsaCvwcOM/MXpLU42dmJslqIZCZXQtcCzBhwgRrb2+vRbQ1oa1jBheMgynz+/WsLMuSk9trEk9/6OzspFyZdnd3A+wMnNLsOu4tn70RjEdl+r1g3LqK68JAdT3QfPRB8WiukMBUoJNg9HtGc8B9kgqjuXbiaA4gPjgmAj+ttZBp0tYxgyWTj0xbjLpQUY2VtAnB4N9kZr+IzqskjTKzFbEyrI7uy4EdErePiW7L2VC5Cu6dAxe9+clSxXrttdc49thjAZ53Hbc8DR/N1WrEUo5ajb6TVCNvvfNbDX0afYXu3nXA42Z2ZcJrOjAJmBx/b0+4nyvpZsKwcE00GjOBrycW9g4FLqxNNpqXLBh+M+OMM85g991353e/+92qhJfruEZkQc+Q3mhu6NCh9Rix9HBaPaZ35q8dsM7qNEKrCZX09PcHTgHmS3o4un2BYAhulXQG8DRwfPT7DXAE0AX8EzgdwMyel/QV4IEY7tLCELEZaOU5w9mzZzNt2jTGjRsHMDbqOXc6bnV8NOdABUbfzO4FVMb74BLhDTinTFzXA9f3R0Cn/hxwwAEEtYGkhWY2IeHtOm4BfDTnFPA3ciug3r38Vh5FNBOtrIfCaO7uu++GOJqTdATB2H9A0mLgkHgNYTT3FGE090PgbAijOaAwmnsAH801HbXZhuI4Tqbx0dzAyMpaTC3xnr7jNIhWHkmkiZdr/3CjnxG84qaLl79TjlarG270+6DVFO44Tr5xo+/kHn+wNy+uu/7jRr8XvEI5tcbrVHPSSnpzo+84jpMj3OhniFbqTTQLaZS569lJEzf6juM0JY1+eLbKw9qNvuM4To5wo1+GtJ7qrdKbaAbSLGvXc3V4+xw4bvQzSCtULKdvXM9OGrjRL4E3RsfJLmm3z7TTrxY3+kU0u0KdynA9O3nFjX5GcaOUD1zPzUkz682NfoZp5oqVZbxcnTzjRj+BG4PWJ4s6zqJMTt80q97c6DuO0zRkzdBmTZ5KcKMfaUblOf0jyzrOsmxO7zSb7tzoZ5xmq1BZpRnKsRlkTJMsl0+WZSvGjT7ZV1hbx4zMy5hlmqnsmknWRuLlUjtyb/S9MjmOUwuaxZYMTluAtGgWBSUpyLxk8pEpS9IcNKOOIcjtOt5AM+mxIOsNE4ekLEl5Gt7TlzRR0iJJXZI6Gp1+K0yVZF3+tHUMMH/5mjSSrRlZrqeN0m+Wy6Av5i9fk1nZG2r0JQ0CvgscDowFTpQ0thFpN3MFKkVW85OmjiG75TJQsmY80tZvs1HQXZZ02Ojpnb2ALjN7CkDSzcDRwMJaJ5SlQq4npfKZ8tSA67gO9JbXBuu7rvptRZ32ZfiXTD6yoVN6jTb6o4GlietlwN7JAJLOBM6Ml92SFjVItor4FAwHnk1bjt7Q5b169yX/26pMPhM6boSespJGH/ouRTU67lO/8EYdH3jggc+R8XZTS/pTNwr6G4Aee6OsjjO3kGtm1wLXpi1HOSTNNbMJacsxULIgfyN03Ih8tkoa9aBYx82aj4GS5fw2eiF3ObBD4npMdHNaB9dxa+P6bXIabfQfAHaVtKOkTYETgOkNlsGpL67j1sb12+Q01Oib2TrgXGAm8Dhwq5ktaKQMNeBhSXMldUtaIelOSRdLWiJJyYCSBktaLekoSe2S1sf7Xo5b3k5PQf66TqukoWNJJxXrBLi7ivi+LOnHFQRtxDRkpqY6q9Bv1fmIbexfUc8rJd0gaWi18daJTOltI8zMjwoP4HxgNfB/gCHAJsAHgW8ALwLtReGPAlYR1k7agWXRXcARwDrg7Wnnq5mP3nRSRZxfBn6cdt6iLIPTliErB7AEOCSevwV4BPha2nI125G6AM1yAFsB3cCHy/hfC1xf5HYr8K143mP0E/6ry8XnR010cgPw1cT1RjoAPk+Yj34ZWAQcDEwEXgVei3E/EsNuT5jGeB7oAj6RiOfLwG3Aj2Nc84HdgAujjpcChxbJfR2wIqb/VWBQ9DsNmA18C3gu+u0C/AFYQ9gRckvaZZ+SvnuMfry+ApgRz/cB/hw7X48UOmDAR4C5RfH8P2B6PN8M+Cbwd0IH7Rpgi2R9AS6IelwBnJ6IpxP4eOL6NODexPU7gFmxziwCjk/4HUHY5vpyrAOfaVQ55v7bO/1gX2Bz4Jdl/KcCx0naAkDSVoQe59TigJLeJOlDhG1dXfURNxf0pZOySHo7YZrivWa2JXAYsMTMfgt8nWBYh5rZu+ItNxMMwPbAccDXJR2UiPKDwDRgGPAQYfrjTYQtjpcCP0iEvYEwytsFeA9wKPDxhP/ewFPASOBrwFeA38W4xwD/3d/8thqSxhBeEOuSNBqYQXhAbgN8Bvi5pBHAr4G3S9o1cftJwE/i+WTCA/rdBH2MBr6UCPsWwkN6NHAG8F1JwyqQbwjB4P8E2I6w9vG9xIts1wGfjHXvnVQxHdlv0n56Z/Ug9PgWEYxyB3AysLLoqf4P4OF4fBxYDJwU/T9B7CUmeg3rCT2RV4DXgfPqJPv1hJ7JY2X8BVwd8/YoMD7t8u6vPqJbj04IPbZbov8coI1gXG8C5hF634uAf8TwuxB69Euj/p6M93YBvycxvUPYrfI68PNE/N+L8bcRRgUvx3iuITwAuoH3xnSfBAw4Nt5vwMWJ+E8E7ol5mBPjmwO0Rf8/EXr4C2Ia64F3R7/OmK9CPdwubV31pbcq4loSy/XlWIZ3AVsTRmzTisLOBCbF8x8DX4rnu8b73xzbwVpg58R9+wJ/i+ftsaznx7KdS2hXhxIM+j+jTobF8KcBz8S8Pg08mIh3EmGk9lw8/zvwSeB9Mf4uQptU3XWSdqXI4gEMig11J2BTwnDxTELvbHBCwd8puu+LwG/j+R+BCxJ+7WyY09+MMHyfXif53weMp7zRPwK4M1b6fYA5aZf5APQxNhqUdYQ1k7OBa2L4EwgPgBuAHwLbR/fTgdcT8S6MjfmFaAj2ifGvBH6dCLd39E/GPzc2/LZoCJIPiUMIBur+GOdggpFaCRwTDcnrwEuETsBL0XicTTD+swt5iPG9JeajYFBWJ9LqBCakraP+6K2K+JawYU7//YRpkV0ID+B/x7IsHGvZ0Dk4Ang8nl9CfEAQeuBWdN8aoDv6t8f6NbxIhlsIHcFOQkfg8uh3JWEqR8B32NDJWxP13U3o4T8FHATcHuN/kPCwuRM4vN568emd0vS8am5mrxKG9tsTeujH9HLfNOBgSfsSGvtNpQKZ2SuE3sk4Sb3FNyDM7I+EyleOo4EbLXAfsLWkUbWWo4aU0sfRwF/YoJOj2TCV9jPC/Pxa4GUzeya6/wuQpM3i9WrCCO04QoM8J8Y/nzCNUuAZYChh3r4Q/1h6358+CPiPWL4F1hN68K8QphAuM7Otzew/zGyPmId7CYboZ4S6JDNbaWafMLPtCfPW20japa9CywDl9FY1ZvYHwkP9m4TR2rRYloVjiJlNjsFnASMkvZswqipM7TxLqBN7JO7bysz62hG0H6GurSVM5R0T3fcnPJCN8OD+F7A7cBbwIwvThWdEeUZG9y6C3bgVuJHe7UtNcKNfmlKvmg8nNNTvRkO9KXCspL/F7Zc7mNkSQqP9KTDLzFaWSyA2gilsPH/YKErlb3QKclRKSXnNbA1RJ4RFs39I2gT4AMHodgFHSNpG0lsIc+uvmtkrcU5/a4Lh+BFh18/6RPwjJb0JwMyWEobyH5W0OcHgb8qG/elbx3T+IOk/o9ugGE8SM7MVhPn59wFtcX1nZ0nvj/l8PgZcR+ghbivpw3EOGzb0Ptcn4v0fSQ9L+mLxtuGUqXc9+zZB138GPijpMEmDJG0et0iPATCz1wgP7G8Q5vxnRff1hBHUtyRtByBptKTDitL5naR58dMSEKZzVhBGiYcQ6souhLn5V2OYOwj29ROE6cHlkt4raXfCgvBRhLWEZYSR3noa1A7d6PcDM5tC2CJ4MWF6RoQpgl+xoZc5lfDdixsriPJ64K2SPlhzYXNCQifbEdYnlhIWaNcSek+PEIbkfyA0+BfirZsRetRvA0YQtnvOi373x9/nJD0Yz5cSGu8zhIXjFwhTeCsIdeE3UY6fEOaLe+NUwpTPyTGenwG9jbTeC8yR9E9CD/FTFj94BpxsZuOA/4zHKX2k3TKY2T8I7exThBHEFwjrbEuBz7KxffsJwUDfFh+oBT5P6BzcJ+klwnrO2xP+q81sPGHR+BxCvSnwLYKR34rQ7ntGfmZWWOM5lNAx6QAuT9y/J2GRuZ3Q4z+5/yUwQBoxt9dsB2F+bWbi+kLgwl7CDwLWpC13kUxtlJ/T/wFwYuJ6ETAqbZmr0Qdh4W7feD6YMHRXvB4D/BXYv5f45xPXaPobf1G4TmACwYg/kXD/ErB0oHmIbt8CvtBLOZ1G0TpT1vXWTAdha+5nku0l6nlRPC/ZrghTSj9IuP8guhXXkY3C1evwnn5p+nzVvGgO/EOEtxObhenAqQrsQ3hgrUhbqF6o5NX/6YRdERDm6O82M5O0NWE7X4eZzS4Ejm9LD4+XDwE7AisGEP8IhW/MI2knwu6Qp2J5viRpnzjlsh8weCB5iHG/CTieMC/+hjzEaa2jgMfKF2PDaepPNkgaImnLwjmh1/4YG+tpEmFBFsq3q5nAoZKGxe2ehxIehsV15NREXPUj7adnVg/Civ9fCbsPLopulwIfiueXEXZcPELYcfGOtGVOyP5TwrTDa4R5wjMIQ8izCjaEMA/+JKGHm8ndH/3Ux+aEedsuwvTMTtH9YsJUz8OJYzs2TOc8GvV4+wDjP5YNWykfBD6YkHkCwUg8SdjNMaA8RL924L6iMinOw1XEl7yycpTKc7MchF1Hj8RjQUJn2xK2iy4mTAdtE93LtivgY2zYEpx8wau4jtR9y2Zh+Os4juPkAJ/ecXqIOx8eknRHvN5R0hyF/0K9JQ7RkbRZvO6K/m2JOC6M7otK7IJwUsZ17GTuT1SSDB8+3Nra2tIWo2LWrl3LkCFD0hZjwIwZM4Zly5a9woZdCJcTvh10s6RrCNNE34+/L5jZLpJOiOE+ovCK+QnAHoT3Gn4vaTcze71cmgUdZ63ssiYP1EYm1/EGsiYP1E6mefPmPWtmI0p6pj1v1tux5557WjNxzz33pC3CgFm6dKkddNBBRthxcAdhfvJZNryB3LMTgzK7TCjanZEMV+4o6DhrZZc1ecyql8l1vDFZk8esdjJR9JG55JHpnr7TOM477zyuuOIKJkzo+Ye3bYEXbcOe5uSLIz0v3ZjZOklrYvjRQPIN1JIvmyjx/6kjR46ks7OT7u5uOjs7a5upKsiaPFC9TJdccgknnXQSd9/d820v13GG5IHGyORGv0a0dczghonZGipWyh133MF2223Hnnvu2ZD0LPH/qRMmTLD29nY6Oztpb2+vW5ptHTMAWDL5yIrC11uegVCNTHfccQd77LEHn/zkJznrrLNqK1gJGq3jgn4hvzquFF/IrQGFCjd/+ZqNKl+zMHv2bKZPn05cP9mJ8DGoqwjf5Cl0DJL/hdrzP6nRfyvC1wMz+f+pzaiTWtPqOnYqx41+HWg2I3PZZZexbNkylixZAuELgHeb2cmE9w+Oi8GKX0Ip9RLRdOCEuPNjR8KLSoVPGjgp4jp2CrjRrxPNZvjL8HngfEldhPnc66L7dYQPgXURvjfTAWDhv1JvJXyP6LeEr1aW3dXhZALXcc7wOf0qaRHjnuRlMzsKwMJHvfYqDmBm/wY+XOpmM/sa4d+eUqeUbto6ZlQ859vCuI5zjPf060gLPhAcx2ly3OgPkLaOGW7UmxTXW+vjOi6PG32nJemr0btRaH5cxwPDjf4A8MrUGrgenTziRt9xHCdHuNF3Wg7vwbc+ruOB40a/zviCr+M4WcKNvuM4LYt3uN6IG/1+4pWotXB9Nh/91ZnreGPc6DcIr3iO42QBN/qO4zg5wo2+01L4iMpxeseNfj9wg9KauF5bH9fxBtzoOw5uFPKA6zjgRr9CalFhvNI5jpM2bvQbjBt+x3HSpE+jL2kHSfdIWihpgaRPR/dtJM2StDj+DovuknS1pC5Jj0oan4hrUgy/WNKkcmk6zkDwB2rr4zqunkp6+uuAC8xsLLAPcI6ksYS/T7vLzHYF7orXAIcT/jdzV+BM4PsQHhLAJcDehH/quaTwoMg6XtHygeu59XEdV2D0zWyFmT0Yz18GHgdGA0cDU2OwqcAx8fxo4EYL3AdsLWkUcBgwy8yeN7MXgFnAxFpmxnEcx+mdfv1HrqQ24D3AHGCkma2IXiuBkfF8NLA0cduy6FbOvTiNMwkjBEaOHElnZ2d/RKwLF4xbV1G4kVtUFjYLeWo1vAfX+riOa0PFRl/SUODnwHlm9pKkHj8zM0lWC4HM7FrgWoAJEyZYe3t7LaIdMKGiVVZMF4xbx5T5fYddcnJ7dULVmKVLl3LqqaeyatUqgD0kfdrMropTcrcAbcAS4Hgze0FB+VcBRwD/BE4rjAbjWs3FMeqvmtlUmohW/VNt17FToKLdO5I2IRj8m8zsF9F5VZy2If6uju7LgR0St4+JbuXcnZQZPHgwU6ZMYeHChRCm73K3btPquI6dApXs3hFwHfC4mV2Z8JoOFHbgTAJuT7ifGnfx7AOsidNAM4FDJQ2LleTQ6JY7sjZMHTVqFOPH92yyWo+v27QcruMNZK39NZpK5i32B04B5kt6OLp9AZgM3CrpDOBp4Pjo9xvCkLCLMCw8HcDMnpf0FeCBGO5SM3u+FplwasqmpLBu093dXdVaR6XrLpVQC3nqQQ1lch23vo7L0qfRN7N7AZXxPrhEeAPOKRPX9cD1/REwTerZI8ji3HF3dzfAzsApjV636ezspJr1m9Nqqav5a7lh4tCq5KkH1ZYRuI57aGEd94W/kesA8Nprr3HssccCPN9s6zZ5H65XiuvYATf6DmBmnHHGGey+++4AqxJevm7TIriOnQJu9FMkK72X2bNnM23aNO6++26AsZIelnQEYd3mA5IWA4fEawjrNk8R1m1+CJwNYd0GKKzbPEATr9vMX74mbRFqiuv4jbSajiulXy9nOa3JAQccQFiKAUkLzWxCwjvT6zZZeXBmHdexU8B7+mVoVEXzCp1dXDetTx517EbfcRwnR7jRL0Een/5OabwutD5507Eb/QyQt0pXK7zcWh/Xce1xo+84jpMj3OgX4T0LpxivE61PnnTsRj8j5KnSOU4leJuoD270E3glcxyn1XGjH3GD7/SG14/WJy86dqOfIfJS6RzHSQ83+k5TksYD0h/KrU8edOxG33H6QR6MQt5pdR270c8YrV7hWgHXUevTyjp2o0/2FJw1ebKGl0/r4zquH7k3+l65nIHg9ab1aVUd597oZ5VWrXCO46RLro1+1g1r1uVLgyyVSZZkaSWyVK5ZkqVW5Nbot6Iyncbj9aj1aTUd59Lot5oS80JW9ZZVuZqRrJZlVuUaCLkz+s2mvLaOGU0ncx5xPbU+raLjhht9SRMlLZLUJamj0ek3K81U2eqh42bJf7PIWQ31asPNUnbNImc5BjcyMUmDgO8CHwCWAQ9Imm5mC+uddrMrCjbkYcnkI1OWpDxp6jgrFNe1LOurv9RLv83WPptZxw01+sBeQJeZPQUg6WbgaKDmBqHZKlF/SOYtg5WtpjpuBT1mXF/9peZt2HXcWBpt9EcDSxPXy4C9kwEknQmcGS+7JS1qkGxV8ykYDjzbyDR1ec2jfHuV9w9Uxw0vu96oly6r1FetZHpbFff2qV9wHVdB3XXcaKPfJ2Z2LXBt2nIMBElzzWxC2nJUg6S59U6jlI6zVnZZkweyKVM5XMcDoxEyNXohdzmwQ+J6THRzWgfXcWvj+m1yGm30HwB2lbSjpE2BE4DpDZbBqS+u49bG9dvkNNTom9k64FxgJvA4cKuZLWikDNUg6SRJcyV1S1oh6U5JBySC9GtaStISSYfUWMxqqWpqrQodZ2ZKT9IS4H9FPa+UdIOkoQn//STdLellSWsk/VrS2IR/u6RldRAt9TKqsg2nLn8RWZMHGiCTzKzeabQEks4HOoCzCBX+VWAi8D4z++wA41wCfNzMfl8rOZ3qSepF0lsI+r7DzC6StC8wC7gIuB7YBDgfOAfY08yektQO/NjMxqQhv+P0Ru7eyB0IkrYCLgXOMbNfmNlaM3vNzH5tZp+VtJmkb0t6Jh7flrRZvHe4pDskvSjpeUl/kvQmSdOAtwK/jj3Kz6WZR6c0ZraSYPTfHZ2uAG40s6vM7GUze97MLgbuA76cjpSOUzlu9CtjX2Bz4Jdl/C8C9iEYhncR9jJfHP0uIGxrGwGMBL4AmJmdAvwd+KCZDTWzK+omvTNgJI0BDge6JL0Z2A+4rUTQWwkvLDlOpnGjXxnbAs/G+cxSr6GfDFxqZqvN7B/An4EvSHoYOJ2wj/ltcXTwJ0t5Tk3S9ZJWS3qsjL8kXR3z96ik8XWUpW6f5ZC0g6R7JC2UtEDSp6P7NpJmSVocf4dFd0m6mrAXfaaktYQ96auBJ4AFhDazdyKNPSXNB74GbCdJFco2SNJDku6I1ztKmhPL4Za4SEocRd4S3edIakvEcWF0XyTpsIR7pj51Umt5StXfvnRaqi5LmhTDL5Y0KeG+p6T58Z6rCzotl0b0G1Bdq7dcJTEzP/o4CHP36wjvNQwCngR2AjYFHgH+DeyRCH8h8Ho83xKYAjwVj45EuCXAISnk533AeOCxMv5HAHcCIoxg5tRJjlJlObaG8Y8Cxif08FdgLGGKpiO6dwCXF+V7CfB/gTnA+4FnCKOyMcDr8XpYvOf+WEanx3pwONAOLOtDtvOBnxDWCiCMFE6I59cA/xXPzwauiecnALfE87GxvDYDdozlOKjeZZoFHZeqvxXodKO6DGwT2+M2wLB4XqxTxXsP7y2NKutaXeUqWX5pVYZmOoCtgLXAcYSpnpkJvwuB54AjEm7fAF4qEc87Cb3Gg+P130jB6Me02yhv9H8AnJi4XgSMqoMMpcrywjrm+XbCFExPfmJjXZTMN/FhXAhHmNb7WwxzL6HHf2L0eyK63wn8KcbRTi9Gn/DwuAs4CLgjNuJngcHF5UJYT9g3ng+O4VRcVoVwjS7TtHRcXH/70mlxuKi/HxTX+aROo3tPuHJpVFPXGi2Xmfn0TiWY2RrgS4QPTR0DPCNpE0mHA/sTnsYXSxohaTjhWySKw7Z7JR0Qh2JrCD3F9THqVYQeUNYo9ar96CZOhzgt8h5C732kma2IXisJay29yfMwMFrSuwg9qV2Bj8TflZK+SjBu36tQ/m8Dn2NDPdgWeNHi9CEbl0OPTNF/TQxfTtaGlWmFNEqe/uq0N/dlJdx7S2MjqqxrdZOrgBv9CjGzKYQh+XHARwmKOReYB8yNx6PAfKATGG1m/4swFTAL6Ab+AnzPzO6J0V5GeFi8KOkzjctNvlDYY/9z4DwzeynpZ6F71Ncay1rgIeBLZnYvMBUYB/wWOIDQwA9g40ZZTpajgNVmNq+/+XAqo0Kd1iWNGtS1usiVxI1+PzCzmwgG/24ze4uZHUmYx/27mX3KzEbF48yEwk8EXjWzIWY2xsy+kojvdjN7q5ltbWbfbHyOytKoV+3rno6kTQiN8CYz+0V0XiVpVPQfRZhy65HHzNosvDtRkGc58LCZHRvDrSfsztqZ8MXJI83ssUJ4M+u08nv09wc+pPAuwM2EKZ6rgK0lFb6FlSyHnjKK/lsRphPLlV3WPpPQKHl61WmJ9HtzH1PCvbc0iG79rmuNkOsNNGpur1UOwrzqU4TFs8LC1B5FYUYlzv83cF/acpfIRxvl5/SPZONFpvvTKssq4xdwI/DtIvdvsPHC1xW95ZuwqPY3wsLasHi+TfQrXlw7oh/ytbNhIfc2Nl7IPTuen8PGC7m3xvM92Hgh9ynComldyzQrOi6uv43Qabk00q5rvclVsuzSqgzNfBBW3v9K2JVwUXS7FPhQPL+MsNj3CHAP8I60ZS6S/6fACuA1wpTEGYQ3jc+K/iKsXzxJmK6a0MiyrGHcBxCGuo8S5uUfjultS1hIXQz8PtGoyuYb+BjQFY/TE+4TgMfiPd8hvuVeoXztbDD6O8VG3UV4AGwW3TeP113Rf6fE/RfFdBcRd3LUu0yzoOMy9bfuOi2XRtp1rTe5Sh3+GQbHcZwc4XP6juM4OSJzf6KSZPjw4TZixAiGDBmSmgxr167NVfrz5s171sxGNCq94cOHW1tbW5/h0tZDKZpVpkbr2MkWmTb6bW1tfPOb36S9vT01GTo7O3OVvqSnG5YYQcdz5/b9Z11p66EUzSpTo3XsZAuf3nEcx8kRbvQrJPlv904+8TrgtAJu9B3HcXJEpuf0s4L38JxCHUjWhRsmZmsR13EqwXv6fTB/+Zq0RXAcx6kZbvR7obiH39Yxw3v9Tg/eIXCaETf6jtMH/qB3Wgk3+o7TC27wnVbDjf4AcEOQDyrRs9cFp9lwo1+GvhqzN/bWxvXrtCpu9B3HcXKEG/0q8N5ga+J6dVoZN/ol8EbvOE6r4ka/CDf4Tn/x9zecZsKNfpV4g3ccp5lwo+84jpMj3Ogn8B67Uw1ef5xmwI2+4zhOjnCj7ziOkyPc6DtOAp+icVodN/qOU0P8oeFkHTf6kWobqzf25sd16OQBN/qO4zg5wo1+DWnWnuLSpUs58MADGTt2LMAekj4NIGkbSbMkLY6/w6K7JF0tqUvSo5LGF+KSNCmGXyxpUjo5chynHG70HQYPHsyUKVNYuHAhwOPAOZLGAh3AXWa2K3BXvAY4HNg1HmcC34fwkAAuAfYG9gIuKTwosk6zPrAdp7/0afQl7SDpHkkLJS3wXmDrMWrUKMaP71HTeoLhHw0cDUyN7lOBY+L50cCNFrgP2FrSKOAwYJaZPW9mLwCzgImNyUV28AeIk2UGVxBmHXCBmT0oaUtgnqRZwGmEXuBkSR2EXuDn2bgXuDehF7h3ohc4AbAYz/RoHFLFG+lGbAq8B5gDjDSzFdF9JTAyno8GlibuWRbdyrlvhKQzCSMERo4cSWdnZ59CdXd3VxRuoFwwbl2/7xm5Rfn76ilrb9S7nJzmp0+jHxv9inj+sqRkL7A9BpsKdBKMfk8vELhPUqEX2E7sBQLEB8dE4Kc1zE/qtHXMYMnkI9MWY0B0d3cD7AycYmYvSerxMzOTZLVIx8yuBa4FmDBhgrW3t/d5T2dnJ5WEGwjhoV9J/2djLhi3jinzS9+35OT26oQaIPUsJ6c16FdNl9RGg3uBjei59NbL6603V45aytuontu6deu48MILAZ43s19E51WSRpnZivjgXh3dlwM7JG4fE92Ws6EjUHDvrKfcWaWZH/5Oa1Ox0Zc0FPg5cF4je4FDhw6te8/ltF6md3rrzZVl/tqaNfhG9NzMjEmTJrH//vszd+7cVQmv6cAkYHL8vT3hfq6kmwlTeGvig2Em8PXE4u2hwIV1Fd5xnH5RkTWTtAnB4N/Uar1An8+H2bNnM23aNMaNGwcwVtLDwBcIxv5WSWcATwPHx1t+AxwBdAH/BE4HMLPnJX0FeCCGu7Qwnec4TjaoZPeOgOuAx83syoRXoRcIb+wFnhp38exD7AUCM4FDJQ2LPcFDo1tL0kwPkwMOOAAz49FHHwVYaGbvNrPfmNlzZnawme1qZocUDHjctXOOme1sZuPMbG4hLjO73sx2icf/pJWnSqmnnpqpDjj5oZKe/v7AKcD82AME7wU6juM0JZXs3rkXUBnvg0uEN+CcMnFdD1zfHwGbGV/Mcxwna+T6jVwffjv1xuuYkzVybfSdfOMG2ckjbvQdp874w8XJErk1+t4QHcfJI7k1+k6+8Ye+k1fc6NcZNy4OeD1wsoMbfSd3uAF28kwujX6jG70bGcdxskIujb7jOE5eyZ3RT6vX7b39bJCmHrwOOFkgd0bfyS9udB3Hjb7jNBR/8Dhp40a/gbR1zPBG7zhOqrjRd3JBlh62WZLFyR9u9B3HcXJErox+VnpYWZHDSQ+vA05a5MboZ62RZU2eViarZZ1VuZzWJjdGP4t4o3ccp9FU8h+5TY8b13zienecN9LyPf2sN3zfxplvXPdOo2l5o98seOOvLc1Uns0kq9P8tLTRb7bG1GzyZpVmLMdmlNlpThpu9CVNlLRIUpekjnql06yNqBWmexql41I0c9m1gu6d7NPQhVxJg4DvAh8AlgEPSJpuZgtrmU4rNJxCHi4Yt472dEXpF43ScZJW0HeSto4ZLJl8ZNpiOC1Ko3fv7AV0mdlTAJJuBo4GqjYIrdbwk5TKW4aNQl11vGTykS2t6wJNpnOniZCZNS4x6Thgopl9PF6fAuxtZucmwpwJnBkv3w48BzzbMCHfyPCcpf82Mxsx0JsHqONFFUSdth5K0awyVaVjp7nJ3D59M7sWuLZwLWmumU1IS568p18PinVcCVksB5fJaUYavZC7HNghcT0mujmtg+vYcTJMo43+A8CuknaUtClwAjC9wTI49cV17DgZpqHTO2a2TtK5wExgEHC9mS3o47Z+TQPUgbyn3y8GqONKyGI5uExO09HQhVzHcRwnXVr6jVzHcRxnY9zoO47j5IjMGv1avsovaQdJ90haKGmBpE9H920kzZK0OP4Oi+6SdHVM+1FJ4xNxTYrhF0ualHDfU9L8eM/VklRCjkGSHpJ0R7zeUdKceM8tceETSZvF667o35aI48LovkjSYfUorzTpKx+9lU2d5ClZd4rCtEtaI+nheHypnjLFNJfE+vawpLkl/MvWYSfnmFnmDsIC4JPATsCmwCPA2CriGwWMj+dbAn8FxgJXAB3RvQO4PJ4fAdwJCNgHmBPdtwGeir/D4vmw6Hd/DKt47+El5Dgf+AlwR7y+FTghnl8D/Fc8Pxu4Jp6fANwSz8fGstgM2DGW0aBal1eW9V6ubOooU8m6UxSmvaDTBpbVEmB4L/4l67AffmS1p9/zKr+ZvQoUXuUfEGa2wswejOcvA48Do2OcU2OwqcAx8fxo4EYL3AdsLWkUcBgwy8yeN7MXgFnAxOj3H2Z2n5kZcGMiLgAkjQGOBH4UrwUcBPysTPoFuX4GHBzDHw3cbGavmNnfgK5YVjUtrxSpJB/lyqYu9FJ3sk65OuzknKwa/dHA0sT1MmrU0OJ0wHuAOcBIM1sRvVYCI/tIvzf3ZX3I+23gc8D6eL0t8KKZrStxT0860X9NDN9fuZqNSvJRrmzqTlHdKWZfSY9IulPSHg0Qx4DfSZoXP2tRTKvUCafGZO4zDPVE0lDg58B5ZvZSsoNoZiapLvtXJR0FrDazeZLa65GGU1+K606R94OE79l0SzoC+BWwa51FOsDMlkvaDpgl6Qkz+2Od03RagKz29Gv+Kr+kTQiN9iYz+0V0XlUY8sbf1X2k35v7mF7k3R/4kKQlhCmLg4CrCEPuwSXu6Ukn+m9F+PBcf+VqNirJR7myqRtl6k4PZvaSmXXH898Am0gaXk+ZzGx5/F0N/JIwNZakVeqEU2OyavRr+ip/nPO9DnjczK5MeE0HCjtwJgG3J9xPjTsg9gHWxGmgmcChkobFnT6HAjOj30uS9olpnZqICzO70MzGmFlbzMvdZnYycA9wXJn0C3IdF8NbdD8h7mDZkdCbvL/W5ZUileSjXNnUhV7qTjLMWwrrCpL2IrSruj2IJA2RtGXhnFAPHysKVq4OO3kn7ZXkcgdh98FfCbs5LqoyrgMIc6CPAg/H4wjCXPBdwGLg98A2MbwIfwTyJDAfmJCI62OEBdQu4PSE+wRCw3sS+A7xbecSsrSzYffOTgSj3QXcBmwW3TeP113Rf6fE/RfFNBaR2CFUy/LKmt6BS4EP9VU2dZKnXN05CzgrhjkXWEDYbXQfsF+dZdoppvVITLdQTkmZytZhP/J9+GcYHMdxckRWp3ccx3GcOuBG33EcJ0e40Xccx8kRbvQdx3FyhBt9x3GcHOFG33EcJ0e40Xccx8kR/x+7gnTm/lQKyQAAAABJRU5ErkJggg==\n"
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "#Check if our distribution is normally distributed\n",
    "\n",
    "hist = results_df.hist(bins=100)"
   ]
  },
  {
   "cell_type": "markdown",
   "source": [
    "Now, to calculate the confidence interval of the ROI, we can use the standard distribution, as we have a large number of simulations. We can use the scipy library to get the confidence interval, using the function norm.ppf() to get the quantiles of the standard distribution. We can also use the describe function to get the statistics of the results:"
   ],
   "metadata": {
    "collapsed": false
   }
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "import scipy.stats as st"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(1.010423544564778, 3.593008469189609)\n"
     ]
    }
   ],
   "source": [
    "# Get the confidence interval of the ROI using the standard distribution\n",
    "interval = (st.norm.ppf(0.025, loc= np.mean(ROI), scale=np.std(ROI)), st.norm.ppf(0.975, loc= np.mean(ROI), scale=np.std(ROI)))\n",
    "print(interval)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The 95% confidence interval of the ROI will be between these 2 values\n",
    "\n",
    "Note that the function ```norm.ppf``` takes the quantile as the first argument, and the mean and standard deviation as the second and third arguments. Since we want to get the 95% confidence interval, we use 0.025 and 0.975 as the quantiles. To get the average and standard distribution of the ROI, we use the numpy functions ```mean``` and ```std```, which will return the empirical mean and standard deviation of the data generated during the simulation.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": "            impacts            CTR       Visitors            CVR  \\\ncount  1.000000e+05  100000.000000  100000.000000  100000.000000   \nmean   5.999030e+05       0.050052   30031.266831       0.499396   \nstd    1.001245e+05       0.010003    7901.093998       0.099647   \nmin    1.627489e+05       0.006749    3111.804727       0.095343   \n2.5%   4.026289e+05       0.030309   15950.632852       0.304002   \n50%    5.997867e+05       0.050051   29532.234129       0.499373   \n97.5%  7.955853e+05       0.069589   46908.820897       0.693940   \nmax    1.045158e+06       0.092695   73104.943275       0.958135   \n\n           Customers       Revenues           Cost            ROI  \ncount  100000.000000  100000.000000  100000.000000  100000.000000  \nmean    14997.328542   74986.642710   22703.421018       2.301716  \nstd      5015.323090   25076.615449    5945.011469       0.658838  \nmin      1425.791553    7128.957767    2472.171271      -0.369106  \n2.5%     6727.525477   33637.627384   12107.334547       1.010177  \n50%     14445.367567   72226.837833   22325.493179       2.301226  \n97.5%   26258.565698  131292.828488   35404.215418       3.587235  \nmax     47847.191767  239235.958836   55099.856282       5.337203  ",
      "text/html": "<div>\n<style scoped>\n    .dataframe tbody tr th:only-of-type {\n        vertical-align: middle;\n    }\n\n    .dataframe tbody tr th {\n        vertical-align: top;\n    }\n\n    .dataframe thead th {\n        text-align: right;\n    }\n</style>\n<table border=\"1\" class=\"dataframe\">\n  <thead>\n    <tr style=\"text-align: right;\">\n      <th></th>\n      <th>impacts</th>\n      <th>CTR</th>\n      <th>Visitors</th>\n      <th>CVR</th>\n      <th>Customers</th>\n      <th>Revenues</th>\n      <th>Cost</th>\n      <th>ROI</th>\n    </tr>\n  </thead>\n  <tbody>\n    <tr>\n      <th>count</th>\n      <td>1.000000e+05</td>\n      <td>100000.000000</td>\n      <td>100000.000000</td>\n      <td>100000.000000</td>\n      <td>100000.000000</td>\n      <td>100000.000000</td>\n      <td>100000.000000</td>\n      <td>100000.000000</td>\n    </tr>\n    <tr>\n      <th>mean</th>\n      <td>5.999030e+05</td>\n      <td>0.050052</td>\n      <td>30031.266831</td>\n      <td>0.499396</td>\n      <td>14997.328542</td>\n      <td>74986.642710</td>\n      <td>22703.421018</td>\n      <td>2.301716</td>\n    </tr>\n    <tr>\n      <th>std</th>\n      <td>1.001245e+05</td>\n      <td>0.010003</td>\n      <td>7901.093998</td>\n      <td>0.099647</td>\n      <td>5015.323090</td>\n      <td>25076.615449</td>\n      <td>5945.011469</td>\n      <td>0.658838</td>\n    </tr>\n    <tr>\n      <th>min</th>\n      <td>1.627489e+05</td>\n      <td>0.006749</td>\n      <td>3111.804727</td>\n      <td>0.095343</td>\n      <td>1425.791553</td>\n      <td>7128.957767</td>\n      <td>2472.171271</td>\n      <td>-0.369106</td>\n    </tr>\n    <tr>\n      <th>2.5%</th>\n      <td>4.026289e+05</td>\n      <td>0.030309</td>\n      <td>15950.632852</td>\n      <td>0.304002</td>\n      <td>6727.525477</td>\n      <td>33637.627384</td>\n      <td>12107.334547</td>\n      <td>1.010177</td>\n    </tr>\n    <tr>\n      <th>50%</th>\n      <td>5.997867e+05</td>\n      <td>0.050051</td>\n      <td>29532.234129</td>\n      <td>0.499373</td>\n      <td>14445.367567</td>\n      <td>72226.837833</td>\n      <td>22325.493179</td>\n      <td>2.301226</td>\n    </tr>\n    <tr>\n      <th>97.5%</th>\n      <td>7.955853e+05</td>\n      <td>0.069589</td>\n      <td>46908.820897</td>\n      <td>0.693940</td>\n      <td>26258.565698</td>\n      <td>131292.828488</td>\n      <td>35404.215418</td>\n      <td>3.587235</td>\n    </tr>\n    <tr>\n      <th>max</th>\n      <td>1.045158e+06</td>\n      <td>0.092695</td>\n      <td>73104.943275</td>\n      <td>0.958135</td>\n      <td>47847.191767</td>\n      <td>239235.958836</td>\n      <td>55099.856282</td>\n      <td>5.337203</td>\n    </tr>\n  </tbody>\n</table>\n</div>"
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# We can also get the empirical 95% confidence interval of all metrics using describe\n",
    "results_df.describe(percentiles=[0.025, 0.975])"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.5"
  },
  "pycharm": {
   "stem_cell": {
    "cell_type": "raw",
    "source": [],
    "metadata": {
     "collapsed": false
    }
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}